Fix uncaught pydantic ValidationError crash on malformed LLM response#255
Fix uncaught pydantic ValidationError crash on malformed LLM response#255Ruanyuxi1337 wants to merge 1 commit into
Conversation
|
Thanks for taking this on. One gap worth covering before merge: this only wraps the async In that sync loop, a malformed structured response from one batch still raises out of So I think the same per-batch isolation should be added to |
keshprad
left a comment
keshprad
left a comment
There was a problem hiding this comment.
Requesting changes because the async retry still aborts the scan after two malformed responses and changes the documented handling of ordinary ValueError. I also agree with @koriyoshi2041 that the synchronous run_batches() path used by semantic_security_discovery remains fail-fast and needs equivalent per-batch isolation. Please add regressions for persistent malformed async responses, ordinary ValueError propagation, and a three-batch sync run with a malformed middle response.
| batch.file_label, | ||
| retry_exc, | ||
| ) | ||
| raise retry_exc |
There was a problem hiding this comment.
A persistent malformed response still reproduces #250 here. Pydantic ValidationError subclasses ValueError, so after the retry re-raises it, the aggregation loop matches it as ValueError and aborts the entire scan. Please handle exhausted Pydantic validation failures as an isolated batch failure while preserving propagation for ordinary configuration ValueError, and add a regression where both attempts are malformed.
| except NotImplementedError as exc: | ||
| # Do not retry on configuration / code errors | ||
| raise exc | ||
| except Exception as exc: |
There was a problem hiding this comment.
This broad catch changes the documented ValueError contract. If the first call raises an ordinary configuration ValueError (for example, a missing API key) and the retry succeeds, arun_batches() returns successfully instead of propagating the configuration error. Please catch Pydantic ValidationError explicitly for retry and let non-Pydantic ValueError bypass retry.
rng1995
left a comment
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Requesting changes. The retry is async-only, catches configuration/programming errors too broadly, and re-raises an exhausted Pydantic validation failure into aggregation, where it can still abort the scan. Isolate malformed-response failures per batch in both sync and async paths while preserving ordinary ValueError/configuration propagation, and add regressions.
| except NotImplementedError as exc: | ||
| # Do not retry on configuration / code errors | ||
| raise exc | ||
| except Exception as exc: |
There was a problem hiding this comment.
Blocking: broad except Exception retries ordinary configuration and programming failures, including unrelated ValueErrors. Catch the intended transient/structured-parse exceptions explicitly so real misconfiguration still propagates immediately.
| batch.file_label, | ||
| retry_exc, | ||
| ) | ||
| raise retry_exc |
There was a problem hiding this comment.
Blocking: after retries are exhausted this re-raises. Pydantic ValidationError is a ValueError, so result aggregation can still abort the whole scan instead of containing only the malformed batch. Return/record an isolated batch failure and add a persistent-malformed-response regression.
| @@ -428,12 +428,39 @@ async def _process(batch: Batch) -> tuple[Batch, list]: | |||
| estimate_tokens(prompt), | |||
There was a problem hiding this comment.
Blocking: only the async path is wrapped. The synchronous run_batches() path used by semantic security discovery remains fail-fast, so a malformed middle response discards earlier successes. Apply equivalent per-batch containment to both paths and test it.
4 participants
Fixes #250. Wraps the per-batch LLM request and parse loop in a try-except guard. If a parsing or network validation failure occurs, it warns, retries exactly once, and falls back gracefully, preventing the entire scan from crashing and losing all report progress.